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ABSTRACT 

We carried out a target survey for Lyman break galaxies (LBGs) and Lyman a emitters (LAEs) 
around QSO SDSS J021 1-0009 at z = 4.87. The deep and wide broadband and narrowband imaging 
simultaneously revealed the perspective structure of these two high-z populations. The LBGs without 
Lya emission form a filamentary structure including the QSO, while the LAEs are distributed around 
the QSO but avoid it within a distance of ~ 4.5Mpc. On the other hand, we serendipitously discovered 
a protocluster with a significant concentration of LBGs and LAEs where no strongly UV ionizing 
source such as a QSO or radio galaxy is known to exist. In this cluster field, two populations are 
spatially cross-correlated with each other. The relative spatial distribution of LAEs to LBGs is in 
stark contrast between the QSO and the cluster fields. We also found a weak trend showing that 
the number counts based on Lya and UV continuum fluxes of LAEs in the QSO field are slightly 
lower than in the cluster field, whereas the number counts of LBGs arc almost consistent with each 
other. The LAEs avoid the nearby region around the QSO where the local UV background radiation 
could be ~ 100 times stronger than the average for the epoch. The clustering segregation between 
LBGs and LAEs seen in the QSO field could be due to either enhanced early galaxy formation in an 
overdense environment having caused all the LAEs to evolve into LBGs, or local photoionization due 
to the strong UV radiation from the QSO effectively causing a deficit in low-mass galaxies like LAEs. 
Subject headings: galaxies: high-redshift — cosmology: observations — large-scale structure of uni- 
verse 



1. INTRODUCTION 

It is well established that QSOs and/or radio galax- 
ies (RGs) are goo d markers fo r gala x y clusters in 
the local universe (|Yee fc Greenl 119871 : [Hall fc Greenl 
1988). Although still very rare, some examples have 
been definitely established of galaxies strongly clustering 
around QSO s/RGs even in the high-z universe beyon d 
z = 3 (e.g.. IZheng et all 120061: lOverzier et all l2006allbl: 
IVenemans et al.ll2005l 12004, 2002), suggesting that such 
fields are indeed special regions of early galaxy formation. 
This trend is naively expected from a generic prediction 
of essentially every model of structure formation because 
the luminous objects tend to sit in the local high-density 
peaks where objects are clustered. A search for high- 
z galaxy clustering around QSOs/RGs thus provides an 
important test of our ba sic idea about the biased galaxy 
formation (|Kaiserlll984l ). 

In the context of the hierarchical galaxy formation, 
it is easy to speculate that the merging process trig- 
gers the activity of the central massive galaxy/AGN. 
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iKauffmann fc HaehneltJ (|2000| ) asserts that a QSO rep- 
resents a brief phase in the early life of every galaxy 
and that the QSO activity is triggered by the major 
mergers that a galaxy experiences during its assembly. 
Such co-evolution of the QSO and galaxy is also sug- 
gested by the tight correlation between the mass of 
the centr al supermassive black h ole and the bulge stel- 
lar mass ( Magorri an et aT1ll998l ) or velocity dispersion 
(|Gebhardt et al.ll2000| ). and that the downsizing trend in 
the luminosity functio ns of AGNs and galaxies are simi- 
lar (jUeda et al.ll2003T ). The overdense region of galaxies 
around high-z QSOs, which deserves to be called a pro- 
tocluster, might be the first site for the co-evolution of 
QSOs and galaxies. 

Beyond z = 3, the detection of star-forming galaxies 
basically relies on two major diagnostics in their spectral 
energy distribution (SED): the redshifted Lya emission 
line and the strong Lyman break in the continuum. Both 
characteristics have been promising tools for identifying 
high-z galaxies, called Lyman a emitters (LAEs) and 
Lyman break galaxies (LBGs), respectively. LAEs have 
often been found to be associated with high-z QSOs/RGs 
or to be tr acing large-scale structures, especially at 2 < 
z < 3 (e.g., JPascarelle et al.lll996l: iPentericci et aill2000l: 
iKurk et atll2000t iPalunas et al.ll2004h . IVenemans et all 
(2005) found 31 LAEs around the luminous radio galaxy 
MRC0316-257 at z = 3.13. Their narrowband (NB) 
imaging and very deep spec troscopy revealed a stru cture 
of at least 3.3 x 3.3 Mpc 2 . iHu fc McMahonl (|1996l ) dis- 
covered two LAEs associated with the QSO BR2237-0607 
at z = 4.55, which was the most distant QSO known at 
the time. LBGs have als o been found to be clustered 
around high-z QSOs/RGs. Zhe ng et al.l (2006) found an 
overdensity of galaxies, whose very red color was consis- 
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tent with that of LBGs, around a radio-loud QSO, SDSS 
J0836+0054 at z = 5.8. The fraction of LAEs among 
their color-selected red galaxies is expect ed to be small 
based o n their SED model simulation. lOverzier et al.l 
(20061) found that LBGs are more strongly clustered 
than LAEs around the most distant kn o wn RG , TN 
J0924-2201 at z = 5.2. lOverzier et "all (|2006bf) con- 
firmed the trend that LBGs are more strongly associated 
with a RG than LAEs in the case of TN J1338-1942 at 
z = 4.1. On the other hand, some discoveries of high-z 
protoclusters have occurred in whi c h no QSO/RG ac- 
tivity is observed. [Shimasa ku et al.1 (|2003l ) discovered a 
clustering region of LAE candidates at z = 4.86 in the 
Subaru Dee p Field, which was not a survey targeted for 
QSOs/RGs. lOuchi et al.l (|2005f) also discovered two over- 
density regions of LAEs at z = 5.7 in the Subaru/XMM- 
Newton Deep Field. Interestingly, no evidence for either 
QSO/AGN or giant Lya nebulae is seen in these struc- 
tures. 

By definition, the Lyman break method should de- 
tect both LBGs without Lya emission and LAEs when 
their continuum fluxes are sufficiently bright. The phys- 
ical and/or evolutionary connection between LBGs and 
LAEs and the cause of the presence or absence of Lya 
emission in the se high-z star-formin g galaxies are still 
open questions. iShaplev et alj (|200lD proposed a plausi- 
ble scenario of two Lya bright phases: a galaxy appears 
as a LAE during its initial starburst epoch when it is 
still dust free, and then becomes a dusty LBG having 
Lya absorption after the ISM metal enrichment. It be- 
comes less dusty, and hence a LAE, again following the 
onset of a superwind when it becomes more than a few 
10 s yr old. To address the connection between LBGs 
and LAEs, we must investigate both populations simul- 
taneously at the same place and at the same redshift : a 
few studies (|Steidel et alj 120001 : lOverzier et al.l l2006al la) 
have been conducted with this motivation in mind in 
high-z protocluster regions. Galaxies in a high-density 
environment are likely to start forming earlier than in 
the general field, and studies of galaxy formation in the 
field only may have missed possible rare active spots as- 
sociated with rich protoclusters. 

In addition, another unsolved issue about the nature 
of high-z galaxies is the environmental effects, especially 
for low- luminosity galaxies. Low- luminosity galaxies are 
closely bound up with the surrounding IGM. Photoion- 
ization heating by strong UV background radiation evap- 
orates the c ollapsed gas in a galact ic halo and inhibits 
gas cooling (|Barkana fc Loebl ll999). This effect is in- 
efficient for bright (L > L*) galaxies residing in deep 
potential wells, but heavily suppresses s tar formation in 
lower mass objects ()Benson et al J 120021) . This could be 
an important feedback process from AGNs to galaxies. 
At z ~ 5, when the reionization of the universe has com- 
pleted, the formation of low luminosity galaxies can be 
suppressed by strong UV background radiation, which 
will be more intense with the presence of nearby strong 
UV ionizing sources such as QSOs. Consequently, the lu- 
minosity function of galaxies around a QSO is expected 
to be flatter than that of the general field. This prob- 
lem, which concerns a low-mass population as a "build- 
ing block" in the high-z universe, can only be addressed 
using 8-10m telescopes. 

We carried out a targeted search for galaxy clustering 



in the vicinity of QSO SDSS J021 1-0009 at z - 5, using it 
as a marker for early galaxy formation sites. We searched 
simultaneously for LAE and LBG clusterings around the 
QSO over a relatively large field of view. Although the 
Lyman break method samples a broader range of redshift 
than NB-imaging, it can be speculated that there is an 
associated structure with the QSO if any o verdensities 
are found around it (e.g. JZheng et al.ll2006l ). We found 
a remarkable contrast between the spatial distribution of 
LAEs and LBGs in the strong UV background field. 

The outline of the paper is as follows. In § 2, we de- 
scribe the observations, data reduction, and LBG/LAE 
sample selection. We compare the clustering trends and 
luminosity functions of the LBG/LAE in fields with and 
without strong UV background sources in § 3. In § 4, we 
discuss possible interpretations of the results. 

Throughout the paper, we analyze in the flat 
ACDM model: £l m = 0.3, n A = 0.7, and H Q = 
70ft.7okms -1 Mpc~ 1 . These parameters are con sistent 
with recent CMB constraints (jSpergel et al.ll2006D . Mag- 
nitudes are given in the AB system. 

2. THE DATA 

2.1. Observation and Data Reduction 

The target field to search for a prot ocluster contain s 
QSO SDSS J0211-0009 at z = 4.874 dFan et al.lfl999h . 
We choose this QSO to coincide its redshifted Lya emis- 
sion line with the central wavelength of a NB filter 
"VB711" (A c = 7126A: SX = 73A). The NB7U nar- 
rowband filter was carefully designed to avoid strong OH 
night sky e mission lines, and was used in the study of 
field LAEs (jShimasaku et al.l l2003h. The imaging d ata 
were taken with Suprime-Cam ( Mivazaki et al.ll2002f ) on 
the Subaru telescope. Observations were made on the 
nights of UT 2004 September and October in Johnson 
V, SDSS i',z', and NB711 filters. Suprime-Cam has ten 
2k x4k MIT/LL CCDs, and covers a contiguous area of 
34' x 27' with a pixel scale of 0."202 pixel" 1 . We could 
sample both LAEs at z = 4.86 ± 0.03 with this NB fil- 
ter and LBGs at z ~ 5 with broadband imaging. The 
broadband imaging also helped to discriminate LAEs at 
z = 4.87 from low-z Ha, [O in] and [O n] emitters. 

There is anoth er QSO, SDSS J0210-0018 (z = 4.77) 
(|Fan et al.l l2001h with only a 10/36 separation from 
SDSS J0211-0009. A single Suprime-Cam's FOV can 
sample both these QSOs simultaneously, although the 
redshift of SDSS J0210-0018 is out of the correspond- 
ing range of the wavelength coverage of iV£>711. Thus, 
we could only sample LBGs in the SDSS J0210-0018 
field. The actual pointing center of the survey field is 
02 ft 10 m 53. s 0, -00°13'44."4 (J2000), which is almost the 
midpoint between the two QSOs. Our original motiva- 
tion for targeting the field was to find a coherent large- 
scale structure of L BGs bridging the two QSOs (e.g., 
Fukugit a~et~aLl I2004D : however we found no evidence of 
such a structure. 

The integration time was 12000-16800 sec in each band. 
For the broadband filters, the typical unit exposure time 
was 900 s for B and 240 s for z'; shorter exposures 
were used for redder filters because of the increase in 
sky brightness with wavelength. For the NB711 filters, 
the typical exposure time was 1800 s. We adopted a com- 
mon dithering circle pattern of a full cycle of dithering 
consisting of 8 pointings. The sky condition was fairly 
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good with a seeing size of 0.5-0.9 arcsec. 

The data were reduce d in almost the same manner as in 
iKashikawa et all (|2004f ), but we used a semi-automatic 
reduction syst em, Distributed Analysis System Hierar- 
chy (DASH) (|Yagi et al.l[2002f ). We performed object 
detec tion and photometry by running SExtractor version 
2.1.6 (|Bertin fe Arnoutill996[ ) on the images. Object de- 
tection was made in the z'-band and 7Vi?711-band images 
for LBG and LAE selections, respectively. For all objects 
detected in a given band pass, the magnitudes and sev- 
eral other parameters were measured in the other band 
passes at exactly the same positions as in the detection- 
band image, using the 'double image mode' of SExtrac- 
tor. We detected objects that had 6 connected pixels 
above 2a of the sky background RMS noise and took 
photometric measurements at the 2a level. The size of 
the final images was 7330 x 9900 pixels (corresponding 
to 24'. 7 x 33'. 3) after removal of very \ow-S/N regions 
near the edges of the images. Object detection and pho- 
tometry is significantly less efficient and reliable close 
to very bright stars due to bright halos and saturation 
trails. A similar degradation occurs near the edges of the 
images because of low S/N ratios. Unfortunately, the 
target field contained some bright stars (see Figure [3]) . 
We carefully defined "masked regions" corresponding to 
these low-quality areas, and removed all objects falling 
within the masked regions. 

Aperture photometry was performed with a 2" diame- 
ter aperture to derive the colors of the detected objects. 
Photometric calibration was made with standard stars in 
the Landolt field SA 92 (|Landoltl fl992l ) for the V-band 
magnitudes, spectroscopic standard stars GD50 for the 
i' and AWll-bands, and FeigellO for the z' band. The 
target field comprised several stars with SDSS (DR4) 
identification and photometry, with which we checked 
the zero-point consistency of the i' and z'-band magni- 
tudes. The 3<7 limiting magnitudes in 2" apertures were 
27.84, 27.18, 26.13, and 26.76 in V, i' , z', and NBlll, 
respectively. The final co-added image had an effective 
area of 742 arcmin 2 . 

2.2. The LBG sample 

The LBG sample was constructed using a color selec- 
tion technique that distinguishes high-z galaxies having a 
strong Lyman break in their SEDs. We used the (V — i') 
versus (i' — z') two-color plane to identify LBGs at z ~ 5. 
The exact selection criteria of the LBG sample were 

V-i'>1.2, 
i' - z'<0.7, 



V -i'>l. 



z') + 1.7. 



(1) 



These criteria were almost the same as lYoshida et al.l 
(2006) except that we did not apply the criterion, B > 3a 
(where 3c means a 3a limiting magnitude), which did 
not greatly affect the sample selection. Therefore, the 
completeness and contamination of the LBG sample 
were expected to be almost the same as evaluated by 
lYoshida et alj (|2006t ). who estimated the completeness of 
their LBG sample to be 60% — 50% at z' = 2 4 15— 26.15. 
According to the estimate bv lYoshida et al.l (|2006f ). this 
Vi'z'-selected LBG sample had a redshift distribution at 
3-7 < z < 5.5 with a peak around z ~ 4.8 and FWHM 
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Fig. 1. — Vi'z' color-color diagram used for the LBG selection. 
All the objects detected down to Ztot = 26.13 (3<r) are plotted. The 
objects (above the red lines) in the upper left were defined as LBGs 
at z ~ 5. The (green) solid curve indicates the model SED track 
(details are described in the text), and the (green) square on the 
track denotes the position of a LBG at z = 4.87. The (blue) solid 
and (cyan) open circles denote the LAAs in the QSO and cluster 
fields, and the (red) solid and (magenta) open triangles indicate 
the LAEs in the QSO and cluster fields, respectively. 

Az = 0.8. Our LBG criteria were incomplete for red 
continuum LBGs (i.e., larger i' — z' color) due to dust 
extinction, while the contamination of galactic stars or 
low-z galaxies are e xpected to be negligible (< 20%) 
(|Yoshida et al.ll2006l ). Figure [1] shows the distribution 
of objects in the (i' — z') vs (V — i') plane; we also 
show the evolutionary track of a model galaxy SED. We 
used the synth etic galaxy SED template s of Bruzal & 
Chariot model (jBruzual fe Char lot! 12003). We assumed 
the Miller & Scalo IMF from 0.1M© to 125M Q , solar 
metallicity Z = Z & = 0.02, ages ranging from 10 6 yr 
to 2 x 10 10 yr, a constant star formation rate, no dust 
extinction, and Madau (1995)'s formula for continuum 
depression due to the intergalactic absorption blueward 
of Lya emission. 

For cases in which the magnitude of an object was 
fainter than the la limiting magnitude, it was replaced 
with the la limiting magnitude. 550 objects down to 
26.13 mag.(3tr) in z' tot , the z'-band total magnitude, met 
our criteria. We adopted the MAG_AUT0 value derived by 
SExtractor for the total magnitude. 

2.3. The LAE sample 

The LAE sample was constructed using a NB-excess 
criterion due to Lya emission and a red color criterion 
due to the strong Lyman break in their SEDs. LAEs 
were selected by the criteria that met both equation |T]) 
and 
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Fig. 2. — V — i' vs i' — NB7U diagram used for the LAE 
selection. All the objects detected down to NBTUtot = 26.76 (3<r) 
are plotted. The objects (above the red lines) in the upper right 
were defined as LAEs at z = 4.87. The (green) solid curve indicates 
the model SED track (details are described in the text) with a 
rest EW of 100A as Ha, [O III], [O II] emitters, and LAEs, and 
the (green) square on the track denotes the position of a LAE at 
z = 4.87. The (blue) solid and (cyan) open circles denote the LAAs 
in the QSO and cluster fields, and the (red) solid and (magenta) 
open triangles idicate the LAEs in the QSO and cluster fields, 
respectively. 



Vi - NB711>0.5, (2) 

where Vi = V x 0.25 + i' x 0.75 is defined as a rough es- 
timate of the continuum flux at 7126A wavelength. The 
second criterion corresponds to a NB-excess with a rest- 
frame equivalent width (EW) of > 10A. Note that the 
LAE sample contained objects that have a fainter z 1 mag- 
nitude than 26.13 mag(3cr), although all the LAE objects 
were detected at above 2a in the z'-band. The first crite- 
rion as in equation ([1]) excludes foreground galaxies with 
emission lines such as Ha at z = 0.09, [O in] at z = 0.42, 
[O n] at z = 0.91, and so on. Figure [3] shows the distri- 
bution of objects in the (V—i') vs (Vi — NB711) plane as 
well as the model SED tracks. We used the same model 
tracks as in LBG selection, although we added emission 
line fluxes with a rest equivalent width of 70 A as Ha, 
[O m], [O n] emitters, and LAEs for each corresponding 
redshift to NB7U. 

As in the case of LBG selection, the magnitude was 
replaced with the la limiting magnitude when it was 
fainter than the la limiting magnitude. 221 objects down 
to NB711 tot — 26.76 (3a) met our criteria. The detec- 
tion completeness as a function of NB711 magnitude was 
estimated in the same way as inlKashikaw a et all ((2006b) 
by counting detected artificial objects distributed on the 
real NB711 image. The detection completeness was thus 
found to be > 90% at NB711 < 25.5 and 50% at the lim- 
iting magnitude NB7W = 26.5. 



2.4. The LAA and LAE classification 

There is a sample overlap between the LBGs and LAEs 
that we selected as above. Some LBGs had Lyman a 
emission lines that also categorized them as LAEs. Some 
LAEs did not meet the criteria of LBGs because their 
z'-band magnitudes were fainter than 3a limiting mag- 
nitude. 

We hereafter distinguish between galaxies with and 
without Lyman a emission, and we define the "Lyman 
a absorbers (LAAs)" as "Lyman break galaxies at z ~ 5 
with almost no Lyman a emission line (EW< 10A) at 
~ 7126A ", and the "Lyman a emitters (LAEs)" as 
"Lyman a emission (EW> 10A) line galaxies at z = 
4.86 ±0.03". Therefore, the overlapping samples (89 ob- 
jects) satisfying both LBG and LAE criteria were all cat- 
egorized as LAEs, and these objects were removed from 
the LAA sample. Strictly speaking, the definition of a 
LAA also includes the galaxy population at z ~ 5 but 
not at z = 4.86 ± 0.03. In the following analysis, we can- 
not distinguish whether we happen to see superposition 
on the sky, but any overdensities at specific regions, like 
the QSO environment or the protocluster of LAEs, are 
speculated to be physically associated with the region. 
Future spectroscopy could enable to clearly determine 
their accurate redshifts; however, most of the LAA/LAE 
galaxies are too faint to be identified with current spec- 
troscopic capability. 

We have 463 LAA samples down to z' tot — 26.13 
(3d), and 221 LAE samples down to NB711 to t = 26.76 
(3a). Based on th e redshift dist r ibutio n of LBGs at 
z ~ 5 estimated by lYoshida et a l. (2006), the expected 
probability of falling within the narrow redshift range 
z = 4.86 ± 0.03 for a LAA sample is 8.8%, which gives 
the expected number of LAAs to have z = 4.86 ± 0.03 
to be ~ 63.95 down to z' tot — 26.0 after correcting for 
completeness. On the other hand, the number of LAEs 
down to z' tot = 26.0 is 103.5 after correcting for detection 
completeness. Therefore, the fraction of NB excess ob- 
jects to all Lyman break selected objects is 61.8%, which 
is larg er than the 20 — 25% evaluated by ISteidel et al.l 
(2000) for LBGs at z = 3.1 down to a similar absolute 
magnitude M ~ —20. The LAE fraction of our estimate 
decreases to 22.6% only when the sample is restricted to 
a bright LAA/LAE sample with z' < 24.5. The differ- 
ence might be caused by either real evolution of the LAE 
fraction among Lyman break selected galaxies, or by the 
LAE sampl e having a fairly h i gh co ntamination of low-z 
interlopers. IShimasaku et al.l (|2006f ) noted from the rest 
UV LF of the z = 5.7 LAE sample that ~ 80% or more 
of the LBG population would have strong Lya emission 
at z ~ 6. 

3. RESULTS 

3.1. Sky distribution 

Figure[3]shows the LAA/LAE distributions in the over- 
all survey field. The distribution of LAEs gives a vi- 
sual impression to have more inhomogeneous structure 
than that of LAAs. We should note that NB searches 
exploring only a small redshift coverage are sensitive 
to the large-scale structure, while the LAA distribu- 
tion is diluted by the sky projection. We focused on a 
12 x 12 arcmin 2 (16 x 16/i 70 1 Mpc 2 in comoving scale) re- 
gion around SDSS J0211-0009, which we call the "QSO 
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Fig. 3. — The LAA (blue circle) and LAE (red triangles) distri- 
butions over the survey field plotted over the i'-band image. The 
small solid square is the QSO SDSS J0211-0009. The solid square 
(orange) is the QSO field, and dashed square (green) is the cluster 
field. The shaded regions are the masked regions around bright 
stars and saturated pixels, in which detected objects were rejected 
due to low S/N. There is a very bright star (mR = 9 mag.) at the 
top of the figure where we had to mask over a relatively large re- 
gion because of ghost features, especially in the NB-image. [See the 
electronic edition of the Journal for a color version of this figure.] 

field". Figure [4] shows the LAA/LAE distributions in 
the QSO field. We also show smoothed surface num- 
ber density contours for LA As and LAEs. The surface 
number density was measured in a circle of 60" radius 
with a Gaussian smoothing of a = 10". The mean value 
and the dispersion of the surface number density in the 
circle for each population were derived from the statis- 
tics of 10,000 randomly chosen positions over the entire 
image. We confirmed that these mean and dispersion 
values for LBGs are almost consistent with those mea- 
sured in a general blank field using the LBG sample at 
z ~ 5 (Kashika wa et al]|2006af ). restricting the LBG sam- 
ple only with z' < 26.31, the same limiting magnitude as 
in this study. The thin and thick contours in Figure [4] 
denote la and 3tr excess from the mean density, respec- 
tively. The distributions of LAAs and LAEs around the 
QSO were in marked contrast with each other. The QSO 
SDSS J0211-0009 lies on a filamentary structure of LAAs 
extending from the northeast to the southwest, while 
the LAEs are distributed around the QSO but avoid its 
vicinity. Although the Lyman break method samples a 
broader range of redshifts than NB imaging, the most 
probable redshift by this I^V-color selection is around 
z ~ 4.8, almost corresponding to the same redshift of 
the NB-excess samples. However, we cannot completely 
rule out the possibility that the filamentary structure of 
LAAs around the QSO is attributed to a mere chance 
of alignment or is incidentally caused by the highly clus- 
tered contaminant populations at law-z, until spectra are 
obtained for the LAAs to confirm that they have similar 
redshifts to the QSO. 
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Fig. 4. — The LAA (blue circle) /LAE (red triangle) distribution 
and its surface number density contours (blue for LAAs and red for 
LAEs) in the QSO field around SDSS J0211-0009 (central orange 
square). The QSO field is defined as a 12 X 12 arcmin 2 region, 
indicated by dashed lines. The thin and thick contours denote 
lcr and 3<r excess from the mean local density. The small mask 
regions are omitted to avoid complexity, except the largest mask 
region shown as a shaded area on the upper right corner. 
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Fig. 5. — Same as Figure[4]but for the cluster field. The cluster 
field is within the dashed lines. The bottom shaded area corre- 
sponds to outside the survey field. 

We found a galaxy overdensity region ~ 9 arcmin east 
of the SDSS J02 11-0009 where both LAAs and LAEs ap- 
parently have strong clustering. We hereafter call this 
region the "cluster field" . This region has a 6<r excess of 
local surface number densities for both LAAs and LAEs. 
The projected size of the overdense region is approxi- 
mately 15/i 7O 1 Mpcx20/i 7 T o 1 Mpc, which is smaller than the 
NB sampling depth ~ SSh^Mpc. No QSOs, AGNs, or 
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Fig. 6. — The cross-correlation functions between LAAs and 
LAEs for the entire (black triangle), QSO (orange circle), and clus- 
ter (green square) fields. 

radio galaxies are known to exist in the cluster field. Note 
that the QSO field and the cluster field slightly overlap 
when an extent of a 12 x 12 arcmin 2 is adopted for both 
fields. Figure[5]shows the LAA/LAE distributions as well 
as their smoothed surface number density contours in the 
cluster field. In sharp contrast to Figure [4] the LAAs and 
LAEs follow a similar distribution in Figure[5l They have 
high number density peaks above 3er significance in al- 
most the same position. The coincidence probability that 
two independent ~ 2' x 2' surface-overdensity regions will 
overlap within 24'. 7 x 33'. 3 is only about 0.001. 

To observe the trend more clearly, in Figure [5] we show 
the angular cross-correlation functions (CCFs) wae{9) 
between LAAs and LAEs for the entire survey field, the 
QSO field, and the cluster fi eld. The CCFs we r e deri ved 
using the same estimator bvlAdclberger 'et alj |2P03) 



wae{0) = 



D A D E - D A R - D E R + RR 
RR ' 



(3) 



where the DaDe, DaR, DeR, and RR denote the 
number of LAA-LAE, LAA-random, LAE-random, and 
random-random pairs having angular separations be- 
tween 9 and 9 + 59, respectively. We generated 100,000 
random points to reduce the Poisson noise in random pair 
counts and normalized DaDe, DaR, DeR, and RR to 
the total number of pairs in each pair count. The ran- 
dom points were created with exactly the same boundary 
conditions as each field avoiding the mask regions where 
saturated stars dominate. The relative amplitude of the 
CCF among different regions was essential in this study, 
irrespective of the absolute amplitude of the CCF for 
each region. For example in the LAA case, the uncer- 
tainty of the CCF due to contamination by foreground 
galaxies was expected to amount to 20% at most, and we 
did not correct the incompleteness of the derived CCF in 



any way. However, the contamination and completeness 
should not be much different among the QSO, cluster, 
and entire fields. We estim ated the Poissonian errors 
only (|Landv fc Szalavlfl993T) for the CCF as 



cr w {9) 




wae{9) 



D a Di 



(4) 



The CCF in the entire field has a slight amplitude ex- 
cess at < 250arcsec (~ 9.3/iyQ Mpc comoving scale), in- 
dicating that LAAs and LAEs generally follow nearly 
identical structures. The CCF in the cluster field shows 
a higher amplitude than that of entire field, suggesting 
that LAAs and LAEs closely coexist in this overdense re- 
gion. On the contrary, the CCF in the QSO field shows 
almost no correlation between LAAs and LAEs on all 
scales, which matches the visual impression in Figure [U 

3.2. Number count 

Figure [7] shows comparisons of number counts between 
QSO and cluster fields. The number counts were investi- 
gated with respect to the z'-band magnitude correspond- 
ing to the rest-frame UV continuum flux (~ 1540A ) 
for LAAs and LAEs, and the ATB711-band magnitude 
corresponding to the Lya flux for LAEs, respectively. 
The number count for each field is denoted as shaded 
regions indicating the ±lcr Poissonian scatter of each 
count. We corrected the com pleteness using the estima- 
tion by lYoshida et al.l (|2006[ ) for LAAs, and the detec- 
tion completeness evaluated in § 2.3 for LAEs, respec- 
tively. We should again note here that no significant 
difference in the completeness between of the QSO and 
cluster fields is expected, even if slight uncertainties oc- 
cur in the estimate procedures. The black regions show 
the number counts ±lcr scatter for the entire field, which 
are regarded as the average number count for each mag- 
nitude and each population. Indeed, the number count 
on the z'-mag f or LAAs in the e ntire field was found to be 
consistent with Yoshid a et al.1 ([2006) for a z'-band mag- 
nitude distribution of field LBGs at z ~ 5. The number 
count on the JVi?711-niag for LAEs in the en tire field was 
also d etermined to be almost consistent with lOuchi et al.l 
(2003) for a N.B711-band m agnitude distribut i on of field 
LAEs at z = 4.86 and with IShimasaku et all (|2004f ) for 
a 7VB704-band magnitude distribution of field LAEs at 
z = 4.79. Note that the sampling field, 12 x 12 arcmin 2 , is 
somewhat larger than the central overdense region in the 
cluster field, and therefore, the absolute number count 
in the cluster field might not be significantly higher than 
the average. 

The number counts of the z'-mag for the QSO and 
cluster fields are almost consistent with each other for 
LAAs, while the z'-mag and NB-va&g counts for LAEs 
differ slightly between the two fields. The z'-mag num- 
ber count of LAEs in the cluster field is higher than in 
the QSO field, although the significance is as small as 
1 — 2a. The difference becomes more significant if we 
take a narrower sampling field for the QSO and cluster 
fields. The LAE NB-m&g number count also shows the 
same trend, but it is not pronounced. These differences 
could be interpreted as that either the number density 
of LAEs is high in the cluster field and low in the QSO 
field, or the Lya (and probably rest-UV continuum) lu- 
minosity of LAEs is high in the cluster field and low in 
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Fig. 7. — The comparisons of number counts between the QSO 
and cluster fields. The number counts are investigated with re- 
spect to the z'-band magnitude for LAAs (upper panel) and LAEs 
(middle panel), and the iV.B711-band magnitude for LAEs (bot- 
tom panel), respectively. Each number count with la Poissonian 
scatter is denoted as a circle (orange) and a square (green) for the 
QSO and cluster fields, respectively. The shaded regions show the 
±ler scatter of the number counts for the entire field. The QSO 
itself is removed from the number counts. 

the QSO field. This trend was not seen in the rest-UV 
magnitudes of the LAA population. A photometric dif- 
ference could exist between LAEs in the QSO and cluster 
fields, although the trend was not sufficiently significant, 
and the sample in this study was so small that the effect 
was not firmly established. We should also note that the 
discrepancy in the number count of LAEs between the 
QSO and cluster fields could be caused by the difference 
of the detection effeciency with 7V_B-band for slightly dif- 
ferent central redshift in each overdensity. 

4. DISCUSSION 

As seen in § 3.1, the relative spatial distribution of 
LAEs and LBGs is in stark contrast between the QSO 
and the cluster fields. Why do LAEs avoid the vicinity 
of the QSO ? We here consider two possible interpreta- 
tions of the observational results. One is that all LAEs 
have already evolved to LBGs in the overdensity region. 
The second is that LAEs cannot form stars due to the 
photoionization effect of strong radiation from the QSO. 

4.1. Evolutionary sequence from LAEs to LBGs 

iShaplev et alj (|2001l ) proposed a plausible scenario of 
two Lya bright phases: a galaxy appears as a LAE 
during its initial starburst epoch when it is still dust 
free, and then becomes a dusty LBG having Lya ab- 
sorption after ISM metal enrichment. It becomes less 
dusty, and hence a LAE again, with the onset of a su- 
perwind as it gets older than a few 10 s yr. The first 
phase of this evolutionary sequence is consistent with 



observational evidence that LAEs are generally younger 
and lower in stellar mass than LBGs, which have ages 
of > 10 9 yr and stellar mass of > 10 10 M^ at z = 3 — 6 
(IShaplev et alJl2001t iStark et all [20061 IE vies et alJ l20M 
lYan et al.l l2006l )~ bverzier et all (|2006bf ) found that 
LAEs around a RG at z — 4.1 are generally younger 
and less stellar-mas sive (a few xl0 8 Af Q ) than LBGs. 
iGawiser et al.1 (|2006l ) also suggested that field LAEs at 
z = 3.1 have a lower stellar mass (~ 5 x 10 8 Af Q ) 
than LBGs. T he stellar mass of L AEs at a higher red- 
shift, z ~ 5 (|Pirzkal et alj I2006D . was also found to 
be much smaller (< 1O 8 M0) than LBGs. Recently, 
iPentericci et al.l (|2007f ) concluded that LBGs with Lya 
emission are much younger and less massive than the 
LBGs without Lya emission bas ed on a systemat i c com - 
parison of spectroscopic sam ple. iVenemans et al.1 (|2005|) : 
iDow-Hygelund et alT {2006) indicate that LAEs have, 
on average, smaller size than LBGs, on the basis of 
HST/ACS high- resolution imaging. The transition from 
LAEs t o LBGs is also suggested in a numerical simula- 
tion by iMori fe Umemural (2006). In addition, the ten- 
dency for an overdense environme nt to favor e arly g alaxy 
formation has been suggested bv iGao et al.l (|2005h . and 
thus, an overdense region i n a QSO field could contain 
a more evolved population. ISteidel et al.l (|2005l ) showed 
a direct evidence suggesting that virialization redshift is 
higher for galaxies in the protocluster region than field 
galaxies. Therefore, one interpretation might be that 
all the galaxies around SDSS J02 11-0009 have already 
passed the first starburst phase as LAEs and are now ob- 
served as LBGs forming a filamentary structure around 
the QSO. 

However, some evidence contradicts this interpre- 
tation. First, in the cluster field, which shows a 
more significant overdensity than the QSO field, LBGs 
and LAEs coexist. The interpretation cannot ex- 
pain the difference of LBG/LAE clustering property 
in the QSO and the cluster fields. Second, if LAEs 
were in previous times clustered around QSOs, some 
examples might be discovered at z > 4.8; how- 
ever, the observational evidence appears to suggest ex- 
actly the opposite. LAEs are usually found clustered 
around QSOs/RGs at z < 3 dPascarelle et a L| 119961: 
IPentericci et alj|2000t IVenemans et al.ll2005D . while only 
a few examples (|Hu fc McMahonl 11996^ ) of them are 
seen around QSOs/RGs at higher redshifts. Instead, 
LBGs are often found around QSOs/RGs at z > 3. 
IVenemans et al.l ()2004f ) found a LAE over density around 
RG TN J0924-2201 at z = 5.2, although lOverzier et all 
(|2006a|) reported that LBG s are more strongly clu stered 
than LAEs in this region. lOverzier et ail JS)06b) con- 
firmed the trend that LBGs are more strongly associ- 
ated with a RG than LAEs in the case of TN J1338-1942 
at z = 4.1. This inconsistency between the trend ob- 
served and a possible evolutionary sequence from LAEs 
to LBGs might be caused by observational bias; NB- 
imaging is difficult for a high-z galaxy survey, which has 
usually been carried out with HST/ACS. More studies, 
especially on estimates of age and stellar mass of LAEs 
at all epochs and their variation with environment, are 
required to validate this hypothesis. 
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As discussed above, LAEs are generally revealed to 
be younger and have less stellar mass than LBGs; how- 
ever, t heir dynamical mass has not been clearly deter- 
mined. lHamana et al.l (|2004f ) estimated a relatively high 
halo mass of LAEs at z = 4.86 of ~ 10 12 M Q , which 
was, however, calculated from their clustering strength 
in a protocluster region. While IShimasaku et alj (|2006fl 
concluded that LAEs at z = 5.7 are distributed almost 
homogeneously in the same survey area. Large uncer- 
tainties still exist in the estimated dynamical mass of 
LAEs against the cosmic variance. In contrast, the dy- 
namical mass of LBGs at z ~ 5 has been well determined 
as 10 11-12 M© based on their clustering strength and the 
UV luminosity - halo mass relation predicted by semian- 
alytic m odel combined with high -resolution N-body sim- 
ulation (Kashi kawa et al.ll2006af h 

It has been recognized that the formation of low 
mass galaxies is suppressed in the presence of an 
external UV field becau s e of photoi o nization (e.g., 
Umemura k Ikeuchil Il984t iRees I 119861: iBabul fc Reed 
19921 lEfstathioul 119921 ; iThoul k Weinberg! Il996fh If 
LAEs are generally less massive than LBGs, their for- 
mation can be prevented to a greater extent in the vicin- 
ity of such radiation sources. This effect may explain our 
finding a different LAE distribution in the QSO field and 
the cluster field. 

We estimated quantitatively the local UV radiation 
strength around the QSO. The local flux density at the 
Lyman limit, i/l, at a radius r from the QSO is 



Aitr 2 



(5) 



where L(i/l) is the QSO luminosity at the Lyman limit. 
For SDSS J02 11-0009, it can be derived by measure- 
ments of the continuum AB magnitude in the rest frame 
1450A corrected for Galactic extinction, AB145 = 19.93, 
and the continuum slope j3 — —0.99 ± 0.70 dFan et ahl 
l200l . assuming F®{v) oc In Figure HI LAEs dis- 
tributed outside r > 770 physical kpc (corresponds to 
> 4.5 comoving Mpc at z = 4.87). The local flux den- 
sity at vl inside a radius of r = 770 physical kpc was 
estimated to be 



F${v L ) = 9.0 x 10~ 19 ergs cm^s - ^" 1 . 



(6) 



This value corresponds to an isotropic UV intensity at 
the Lyman limit of 



J21 ~ 72, 



(7) 



when the UV intensity assumes a power-law spectrum 
of J(v) — J2\{yjvi,) a x 10~ 21 ergs cm -2 s _1 Hz -1 sr _1 . 
The intensity of the background radiation at z — 4.5 was 
evaluated to be \og(J(v>L)) ~ —21.2 by the QSO prox- 
imity e ffect measurements o f ICooke. Espev. k CarsweSI 
(| 19971) ; IWilliger etlrtl (|1994f ). Assuming that the back- 
ground radiation at z = 4.87 is almost the same as this 
value, the relative strength of the local QSO ionizing ra- 
diation was estimated to be ~ 100 times the background 
radiation at LAEs located 4.5Mpc distant from the QSO. 
Therefore, the local UV radiation is actually enhanced 
by the QSO at the position where the deficit of LAEs is 
observed in the QSO field. 
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Fig. 8. — The delay time of star formation, tdclay, as a func- 
tion of the radiation intensity, J21, for a given virial mass , M vlr , 
according to the hydrodynamical model of Kitayama ct al. (2000, 
120011) . The delay time is counted from the time when 1/10 of the 
baryon mass of each halo was cooled without UV radiation. The 
local radiation density in the region where LAEs are deficit around 
the QSO corresponds to J21 > 72. 

We have investigated how this locally enhanced UV 
radiation influences the formation of LAEs by means 
of radiation-hydrodynamic simulations. The simula- 
tions are an improve d vers ion of those described in 
iKitavama et alj (|2000l 1200 If ) with an addition of he- 
lium atoms in the chemical reaction network. Under 
the assumption of spherical symmetry, they solve self- 
consistently the dynamics of dark matter and baryons, 
non-equilibrium chemistry of nine species (e, H, H + , H _ , 
He, He + , He ++ , H2 and H^~), and the radiative transfer 
of ionizing photons. Self-shielding of H2 against photo- 
dissociation is also taken into account. As an initial den- 
sity contrast for the dark matter and baryons, we adopt 
a spherical top-hat perturbation that has just turned 
around from the Hubble expansion at z = 8.3 and is 
destined to collapse (in the absence of thermal pressure) 
at z = 4.87. We assume for simplicity that the gas is 
exposed to constant and isotropic external UV radiation 
with a spectral index a — — 1, which is close to the ob- 
served mean for SDSS J02 11-0009. In order to cover a 
wide range of the UV source distance, we vary the UV 
intensities over the range J21 = 0.1 ~ 10 4 . Given large 
uncertainties as to when the radiation source turns on, 
we first explore the case in which the gas has been ex- 
posed to the UV radiation since the turn-around epoch, 
z = 8.3; the impact of photoionization is likely to be 
maximal in this case. We will also discuss separately the 
case in which the radiation source turns on later. 

In general, the enhanced UV radiation leads to a con- 
siderable delay in star formation in low-mass halos. Fig- 
ure [5] illustrates the delay time of star formation as a 
function of radiation intensity for a given virial mass of a 
halo, M v i r . The delay time is defined as the time between 
the epochs at which 1/10 of the baryon mass in a halo 
has cooled with and without the external UV radiation. 
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The gas is regarded as "cooled" when its temperature 
drops below 1000K or its density exceeds 10 6 cm~ 3 . Fig- 
ure [5] shows that star formation is completely suppressed 
in a halo with M v i r < 10 9 M Q under the background UV 
field with J21 ~ 1. In contrast, it remains almost un- 
affected in a massive halo with M v - iY > 10 n MQ even in 
the vicinity of radiation sources. This is mainly because 
photoionization can raise the gas temperature only to 
several times 10 4 K. In an intermediate mass halo, the 
star formation is suppressed during the initial stage of 
the collapse and the delay time depends sensitively on 
the radiation strength. For example, the UV intensity 
corresponding to equation (O will suppress the star for- 
mation for ~ 10 8 yr in a halo with Af v ir = 1O 1O M0. Such 
a halo will be able to host bright galaxies ~ 10 8 yr later 
than those placed in the weaker UV field. 

The above results imply that if the observed LAE de- 
ficiency around the QSO is due to photoionization and 
the age of the LAE is 10 6 - 10 s vr fGawiser et aQ 120061 : 
lOverzier et"aT1l2006bl : IPirzkal et al.ll2006h . the LAE mass 
is M v j r ~ 1O 1O M0. This is so mewhat smaller than pre- 
vious observational estimates (|Hamana etalll200l . As 
mentioned already, the dynamical masses of the LAEs 
are still poorly determined from observations and we do 
not consider this difference to be critical. Our results fur- 
ther suggest that photoionization cannot delay the star 
formation in massive LBGs with M v ; r ~ 10 11-12 Mq. 

Note that our model is restricted to the case in 
which stars form at the center of a nearly spherical 
halo. If star formation takes p lace after a disk-like col- 
lapse or in substructure s (e.g.. iSusa fc Umemura I [20001 
iMori fe Umemural l2006f ) . the impact of photoionization 
will be greater and the inferred mass of the host halo can 
be larger. 

Another important aspect of the photoionization effi- 
ciency by a local ionizing source is the time difference 
between the QSO activity phase and collapse time of 
galaxies. Even low-mass galaxies could collapse if they 
form before the QSO activity has grown. To examine 
this possibility, we also performed runs in which halos 
that have already collapsed at z = 4.87 with a gas den- 
sity profile of cx r~ 2 are exposed to an external UV field. 
In this case, star formation cannot be delayed even in a 
halo as small as M V - 1T — 10 9 M Q . Hence, another possible 
interpretation of the observational result is that LAAs 
have already formed, or are in the process of formation, 
at the time of the active phase of the QSO host galaxy, 
whereas LAEs are young, having formed much later than 
the QSO/LAA formation. This scenario is supported by 
previous results that the stellar mass of LAEs is smaller 
than LBGs and is almost unchanged at all the epochs 
(lOverzier et alj|2006bt fcawiser et aLll2006t |Pir~zkal et all 
12006ft . although more accurate estimates of the stellar 
mass of high-z LAEs should be accumulated in the fu- 
ture. This picture is also consistent with the idea that 
a galaxy first appears as a LAE during it s initial dust- 
free starburst phase (Sha plev et al.l l2~001h . This inter- 
pretation also requires accurate estimates of the forma- 
tion epochs of LAAs/LAEs relative to the evolutionary 
timesc ale of QSO activity, which have not yet been deter- 
mined. iKawakatu. Umemura. fc Moril (|2003l ) proposed a 
QSO evolution model with a prediction of a timescale 
for a Q SO active phase to be ~ a few 10 s yr. IShen et all 
(|2007| ) estimated the timescale of the luminous accre- 



ation phase to be ~ a few 10 7 — 10 8 yr for QSOs at z > 3.5. 
Th ese timescales are co mparable to the LAE age ~ 10 8 
yr dGawiser et al. 2006) and is shorter than the LBG age 
> 10 9 yr (Shaplev et al.ll2~001h estimated by multicolor 
S EP fitting. 

lOverzier et al.1 (|200 6b) concluded that LAEs prefer re- 
gions that are devoid of UV-bright LBGs in their pro- 
tocluster region at z = 4.1. Their result is inconsistent 
with ours in the cluster field, where we found that LBGs 
and LAEs coexist in an overdense region. The LAEs in 
their study might not have avoided LBGs but rather the 
much brighter central radio galaxy, which has a UV lu- 
minosity of 6i* =4 , although the photoionization effect 
was less significant than for QSOs. 

The importance of radiative feedback from a local 
strong UV ionizing source is suggested by several lines 
of observational evidence. The QSO proximity effect 
emerged in nearby QSOs, where the relative strength 
of the local QSO ioni zing radiation is only > 0.1 times 
the background (e.g.. iBaitlik. Duncan fe Ostrikerl 119881 : 
iLu. Wolfe, fc Turnsheklll99lD . which is much more sen- 
sitive to the QSO inten sity than we found in this study. 
lAdelberger et all (|2003h found that LBGs are associated 
with H 1 underdensities at a separation of < 0.5/i _1 Mpc, 
based on the statistics of several LBG samples and IGM 
absorption lines in the spectra of background QSOs. 
This result was interpreted as being caused by cither 
photoio nization from LBGs or by the LBG galactic 
wind. iFrancis fc~B land-Hawthorn (2004J) detected no 
Lya emission within IMpc of a luminous QSO at z = 2.2, 
and the neutral hydrogen clouds near the QSO may have 
been photoevaporated. Along with these examples, the 
observational result suggests that the photoionization ef- 
fect from a local strong UV ionizing source could play an 
important role in making a possible "habitat segrega- 
tion" between LAAs and LAEs in the high-z universe. 

5. CONCLUSIONS 

In this paper, we have presented a possible cluster- 
ing segregation between LAAs and LAEs around a high- 
z QSO. We carried out a wide-field target survey for 
LAAs and LAEs simultaneously around a QSO SDSS 
J0211-0009 at z = 4.87. Our main conclusions can be 
summarized as follows: 

1. LAAs form a filamentary structure including the 
QSO, while LAEs distribute around the QSO but avoid 
its vicinity. 

2. LAAs and LAEs are spatially cross-correlated in a 
protocluster field where no strongly UV ionizing source 
such as a QSO or radio galaxy is known to exist. 

3. We found a weak trend that number counts based 
on Lya and UV continuum fluxes of LAEs in the QSO 
field are slightly lower than in the cluster field, whereas 
number counts of LAAs are almost consistent with each 
other. 

4. The LAEs avoid the region around the QSO, where 
the UV background radiation could be 100 times greater 
than the average intensity at the epoch. The clustering 
segregation between LAAs and LAEs seen in the QSO 
field might be caused by either all the LAEs having al- 
ready evolved into LAAs in the QSO field, or photoion- 
ization around the strong UV source effectively causing 
a dearth of low-mass galaxies such as LAEs. 

These results can only be achieved by wide-field imag- 
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ing targeting QSO/RG fields. For example, in the J0211- 
0009 field, small FOV imaging revealed only the filamen- 
tary structure of LA As around the QSO, without show- 
ing the larger surrounding structure formed by LAEs (it 
"fails to see the forest for the trees"). Recently, simi- 
lar large FOV imaging revealed a cosmic web of LBGs 
aroun d the RG TN J1338-1942 at z = 4.1 (jlntema et al.l 
2006). Although our sample cannot, in isolation, allow 
the derivation of a general picture of the properties of 
high-z galaxies in an overdense field around known high- 
z objects, it provides a potentially important perspec- 
tive for understanding the nature of high-z galaxies in 
different environments. Future comparative studies of 
different fields and different UV background fluxes (dif- 



ferent QSO/RG luminosity) would permit more quanti- 
tative discussions. 
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